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14.  ABSTRACT 

The  ultimate  goal  of  this  project  is  to  create  a  user-friendly  and  effective  method  to  assess  surgeon  readiness,  after  new 
training  and  after  prolonged  periods  without  training  or  practice,  to  enhance  patient  safety.  In  this  project,  a  FLS  trainer  is 
instrumented  with  motion  tracking  and  sensor  data  capture.  The  data  captured  is  automatically  analyzed  by  our  software 
algorithm  to  provide  objective  skill  assessment  and  performance  feedback  to  the  surgeon.  Progress  on  the  laparoscopic 
simulator  development  and  testing  is  reported  here. 

15.  SUBJECT  TERMS 
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INTRODUCTION 

The  ultimate  goal  of  this  project  is  to  enhance  patient  safety  by  creating  an  objective, 
user-friendly  and  highly  effective  technology  platform  to  assess  surgeon  readiness  after 
new  training  and  after  prolonged  periods  without  training  or  practice.  A  real-time 
analysis  of  individual  performance  data  could  be  used  to  detect  physical  fatigue, 
impairment  or  other  surgeon-factors  that  may  threaten  patient  safety.  Our  integrated 
plan  involves  the  development  of  automatically  graded  surgical  tasks  based  on  the  FLS 
skills  curriculum.  In  this  project,  a  FLS  trainer  is  instrumented  with  motion  tracking  and 
sensor  data  capture.  The  data  captured  is  automatically  analyzed  by  our  software 
algorithm  to  provide  objective  skill  assessment  and  performance  feedback  to  the 
surgeon.  This  concept  of  an  integrated  training  program  with  state-of-the  art  simulation 
technology  will  ultimately  lead  to  real-time  surgeon-factors  assessment  and  surgeon- 
specific  reset  training  to  enhance  military  surgeon  readiness,  improve  outcomes  and 
increase  patient  safety. 


BODY 

This  project  was  funded  as  of  21  July  2008.  The  contract  expiration  date  was  31  March 
2012  (research  ending  29  Feb  2012). 

A  final  report  for  this  project  is  attached  with  emphasis  on  progress  made  during  the 
period  August  201 1  -  Feb  2012  is  described  below. 

A  no-cost  extension  of  time  for  the  original  one-year  project  was  granted  on  June  18, 
2009.  The  approved  revised  statement  of  work  as  is  presented  in  Table  1.  An  additional 
no-cost  extension  of  time  and  additional  funding  for  a  half  time  engineer  was  requested 
08  October  2010.  During  the  period  of  consideration  of  the  funding  request,  data 
collection  continued  but  analysis  and  engineering  work  was  delayed.  The  no-cost 
extension  of  time  and  additional  funding  was  approved  by  contract  modification  on  03 
May  201 1 ,  with  a  final  contract  extension  of  time  approved  on  25  Jan  2012.  The  work 
for  the  additional  funds  was  directed  toward  Task  4  with  the  addition  of  integrating 
augmented  reality  feedback  with  the  FLS  trainer. 


TABLE  1 :  STATEMENT  OF  WORK:  Deliverable  tasks 

1. 

Describe  a  methodology  to  define  and  objectively  measure  basic 
laparoscopic  skills  in  surgeons  (based  on  the  FLS  curriculum). 

2. 

Provide  preliminary  data  demonstrating  “proof  of  concept”  that  a 
physical-based  laparoscopic  simulator  can  reliably  differentiate 
between  novice  and  expert. 

3. 

Describe  a  methodology  and  provide  preliminary  data  to  document 
individual-specific  laparoscopic  skill  deterioration  during  prolonged 
absence. 

4. 

Describe  a  methodology  to  employ  augmented  reality  technologies 
to  support  individual-specific  retraining  of  basic  laparoscopic  skills 
following  prolonged  absence.  To  integrate  augmented  reality 
feedback  during  the  training  process. 
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In  previous  annual  reports,  initial  progress  on  Tasks  1, 2  and  4  were  reported.  These 
reports  described  steps  taken  to  determine  the  scope  and  design  of  simulated  tasks, 
development  of  the  instrumented  simulator,  refinement  of  task  metrics,  and 
incorporation  of  augmented  reality  technology.  Technology  development  time  and 
funding  limitations  have  limited  the  ability  to  explore  the  impact  of  prolonged  absence  on 
skill  degradation  on  subjects  that  have  reached  proficiency  (Task  3).  This  report 
provides  an  abstract  of  simulator  development  and  task  metric  measurements  below  for 
orientation  to  the  newly  reported  data  analysis. 

TASK  1:  Describe  a  methodology  to  define  and  objectively  measure  basic 
laparoscopic  skills  in  surgeons  (based  on  the  FLS  curriculum) 

Determine  scope  and  design  of  simulated  tasks 

Work  was  largely  completed  on  this  task  during  the  2009  reporting  period. 

The  design  of  our  instrumented  simulator  box  was  based  on  the  SAGES  FLS 
laparoscopic  trainer  box.  Like  the  SAGES  FLS  curriculum,  the  simulator  curriculum  is 
proficiency-based,  whereby  trainees  are  oriented  to  the  materials  and  self-practice  until 
expert-derived  performance  levels  are  reached.  Like  the  FLS  trainer  box,  the 
instrumented  simulator  allows  practice  of  technical  skills  to  improve  dexterity  and 
psychomotor  skills.  Key  technical  skills  identified  for  training  during  simulated 
laparoscopic  manipulation  include: 

•  Eye-hand  coordination 

•  Ambidexterity 

•  Depth  perception 

•  Complimentary  use  of  both  hands  (bimanual  skills) 

•  Familiarity  with  common  laparoscopic  surgical  devices 

•  Instrument  navigation  and  transferring 

•  Accuracy  of  instrument  placement 

•  Proper  application  of  force  appropriate  to  task 

The  skills  testing  tasks  that  were  the  most  amenable  to  study  for  implementation  with 
our  instrumented  simulator  include  pegboard  transfer,  pattern  cutting,  and  intracorporeal 
knot  tying.  These  tasks  were  chosen  based  on  variety  of  instrumentation  used  and  skill 
types  tested,  ease  of  implementation  during  the  project  time  constraints,  and  the  ability 
to  incorporate  sensors  on  instruments.  For  each  task,  the  metrics  to  be  measured  were 
modeled  after  those  scored  in  the  FLS  curriculum  tests,  and  include  timing  of  each 
exercise,  accuracy  and  efficiency  of  performance,  and  error  tracking.  In  addition,  other 
metrics  are  being  calculated  which  can  be  used  to  deconstruct  specific  movement 
patterns  of  the  participants. 

Instrumented  Simulator  Development: 
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Significant  time  was  invested  into  developing  a  graphical  user  interface  for  the  FLS  so 
that  anybody  could  setup  the  simulator  and  test  subjects  without  knowledge  about  the 
software.  Instructions  were  included  with  the  software  and  given  to  experimenters  at 
Wayne  State  University  (WSU)  and  Beaumont  Hospital  for  data  collection. 

We  have  developed  software  to  automatically  gather  position  and  orientation  data  from 
magnetic  trackers  attached  to  the  laparoscopic  tools  during  subject  trials.  Figure  1 
shows  the  laparoscopic  tools  with  the  attached  sensors  (trakStar,  Ascension 
Technology  Corp.)  that  allow  for  the  tracking  of  the  tooltips  and  also  of  the  gripper  state. 
Internal  to  the  FLS  trainer  is  a  camera  and  LED  illumination  source  as  well  as  a 
magnetic  transmitter.  The  magnetic  transmitter  is  placed  in  the  far  left  corner  of  the 
trainer  housing.  The  transmitter  does  not  interfere  with  the  work  envelope  and  is  out  of 
view  of  the  camera.  The  equipment  is  placed  on  a  plastic  table  to  minimize  interference 
with  the  magnetic  tracker. 


Figure  1:  FLS  trainer  with  magnetic 
sensors  on  the  tool  shaft  and 
handles.  This  allows  tooltip  tracking 
and  gripper  state. 


Each  laparoscopic  tool  is  fitted  with  two  magnetic  trackers  that  determine  the  position  of 
the  instrument  in  space  and  if  the  trigger  is  open  or  closed.  Figures  2  a  and  b  illustrate 
how  the  sensors  attach  to  the  laproscopic  tool  using  a  mounting  bracket.  Currently  the 
handle  sensor  is  attached  with  a  wire  tie  and  the  shaft  sensor  is  held  in  place  using  two 
machined  brackets  as  seen  in  figure  2b.  Figure  2c  illustrates  the  mounting  bracket  (half) 
CAD  rendering. 


Figure  2:  a:  Flandle  Sensor  b:  Figure  3  Tool  Sensor  c:  Mounting  Bracket 

The  complete  system  consists  of  a  the  FLS  trainer  with  magnetically  instrumented  tools, 
Magnetic  Tracker,  CRT  monitor,  and  a  PC.  Figure  3  illustrates  the  equipment  setup. 
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Figure  3:  FLS  Laboratory  Setup 


CRT 


Magnetic  Tracker 


Computer 


FLS  Trainer 


Instrumented 
Laparoscopic  tools 


Software 

Figure  4  is  a  screen  capture  of  the  software.  The  software  was  written  in  Matlab  version 
2009.  During  normal  operation  there  are  two  forms  of  interest.  The  first  form  or  controls 
form  consists  of  a  window  that  presents  the  user  with  operation  instructions  and  allows 
the  user  to  begin  data  collection.  The  controls  form  also  contains  menu  items  to  initialize 
the  magnetic  tracker  and  calibrate  the  system  sensors.  Generally  the  software  is  run 
with  one  user  performing  the  laparoscopic  tasks  an  observer  calibrates  the  instrument 
and  triggers  the  recording.  This  can  eventually  be  handled  by  one  person  with  the  use 
of  a  foot  switch  but  this  feature  has  not  been  implemented  yet. 


ile  the  task  is  running  (and  the  figure  is  active): 


-  Press  "R"  each  time  the  tip  of  the  right  tool 
is  touched  to  the  top  of  the  6  pegs  on  the 

right  half  of  the  peg  board  to  record  peg  positions 

-  Press  “L“  each  time  the  tip  of  the  left  tool 
is  touched  to  the  top  of  the  6  pegs  on  the 

left  side  of  the  peg  board  to  record  peg  positions 


Figure  4:  Software  Screen  Capture 
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The  controls  form  has  a  menu  items  for  initializing  the  magnetic  tracker,  calibrating  the 
sensors  for  each  task.  The  allowable  tasks  include  Peg  Transfer,  Circle  Cutting,  and 
Intracorporeal  Suture.  The  data  collection  menu  item  contains  a  sub  menu  for  starting 
and  ending  recording  tasks  for  all  three  tasks.  The  collected  data  is  saved  in  a  tab 
delimited  text  file.  The  data  in  the  file  consists  of  25  values  per  sample.  The  data 
structure  includes  a  timestamp,  Left  tool  tip  (x,y,z, roll, pitch, yaw),  Left  Handle 
(x,y,z, roll, pitch, yaw),  Right  tool  tip  (x,y,z, roll, pitch, yaw)  and  Right  handle 
(x,y,z, roll, pitch, yaw).  Each  tool  and  handle  is  therefore  represented  by  six  degrees  of 
freedom  and  from  this  information  the  positions  of  the  tool  can  be  reconstructed.  Data 
gathering  from  this  software  is  used  to  assess  performance  and  feed  into  the 
augmented  reality  software  kernel. 

The  software  development  also  includes  the  calibration  of  the  tool,  sensors  and  the  task 
platform  (e.g.  peg  board  for  the  peg  transfer  task  and  the  circle  for  the  pattern  cutting 
task).  The  system  will  automatically  capture  the  tool  position  and  orientation  and 
computes  new  metrics  such  as  efficiency  of  movement  and  tool  trajectory.  The 
automatic  data  collection  is  important  because  all  of  the  objective  metrics  will  be  derived 
from  them.  The  tool  trajectory  data  is  stored  and  can  be  replayed  as  an  animation  as 
shown  in  Figure  5.  This  has  an  added  advantage  over  the  video  file  since  you  can 
change  the  perspective  to  any  view  and  trace  where  the  instruments  have  travelled. 
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Figure  5:  Left:  3D  animation  of  the  task,  including  instrument  traces.  Right:  Analysis  of 
tool  tip  height.  The  system  uses  this  data  to  compute  the  task  metrics. 

Task  Metrics 


In  this  2010  reporting  period,  significant  refinement  was  made  to  the  task  metrics 
analysis.  Table  2  has  a  list  of  metrics  that  are  computed  for  the  peg  transfer  task. 
Moreover,  some  of  these  primary  metrics  can  be  further  broken  down  into  subtasks 
such  as  picking  up,  passing  and  placing  pegs  as  well  as  performance  on  any  one 
individual  peg.  Data  can  be  broken  down  by  the  subject’s  non-dominant  vs.  dominant 
hand  performance  (to  determine  degree  of  ambidexterity).  We  can  also  categorize 
errors  into  subgroups  such  as  missed  grasps  and  dropped  pegs. 
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TABLE  2:  FLS  Task  1  (Peg  Transfer)  Metrics 

• 

Time 

• 

Best  Possible  Time 

• 

Degree  of  Ambidexterity 

• 

Pathlength 

• 

Economy  of  Motion 

• 

Reversals 

• 

Working  Volume 

• 

#  of  Pegs  done  out  of  sequence 

• 

#  of  Pegs  done  in  parallel 

• 

Parallel  Ratio 

• 

Errors 

• 

Secondary  Task  Performance 

Similarly,  for  the  pattern  cutting  task,  metrics  to  measure  the  performance  of  the  task 
are  calculated.  Image  processing  software  was  developed  (Figure  6)  that  grades  the 
‘quality’  of  the  cut  circle  and  gives  the  subject  feedback  as  to  how  well  the  pattern  was 
cut  and  what  errors  were  made. 

Before  the  task  starts,  the  back  of  the  gauze  is  marked  at  the  6  o’clock  position  for 
orientation  purposes.  After  the  task  is  completed,  the  back  of  the  cut  circle  is  marked 
with  the  subject  ID  and  trial  number.  For  calibration,  we  take  a  photo  of  an  uncut  circle. 
We  then  take  a  photo  of  the  cut  circle  for  the  image  processing  analysis.  Our  computer 
program  automatically  measures  the  undercut  and  overcut  regions  and  breaks  it  down 
by  quadrants. 


Figure  6: 

Top  Left:  Digital  photo 
of  the  cut  circle. 

Top  Right:  Image 
processing  detects 
outline  of  the  circle. 


Bottom  Right: 

Overcut  Regions 
broken  down  by 
quadrant. 

Bottom  Left:  Undercut 
Region  broken  down 
by  quadrant. 
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Table  3  has  a  list  of  metrics  computed  for  the  circle  cutting  task.  Moreover,  some  of 
these  primary  metrics  can  be  further  broken  down  by  tool  (grasper,  scissors)  or  by 
sectors  of  the  circle. 


TABLE  3:  FLS  Task  2  (Pattern  Cutting)  Metrics 

• 

Time 

• 

#  of  Tool  Exchanges 

• 

Pathlength 

• 

Economy  of  Motion 

• 

Reversals 

• 

Working  Volume 

• 

#  of  Cuts  Made 

• 

Overcut  Region 

• 

Undercut  Region 

• 

Reversals 

• 

Scissors  Tool  Angle 

• 

Errors 

• 

Secondary  Task  Performance 

TASK  2:  Provide  preliminary  data  demonstrating  “proof  of  concept”  that  a 
physical-based  laparoscopic  simulator  can  reliably  differentiate  between  novice 
and  expert 


In  the  2010  and  201 1  reporting  periods,  data  was  collected  from  47  subjects  in  the  peg 
transfer  and  circle  cutting  tasks.  A  total  of  1388  trials  of  the  peg  transfer  task  and  214 
trials  of  the  circle  cutting  task  were  analyzed.  Analysis  of  the  novice  learning  curve  and 
comparison  with  expert  data  within  this  subject  group  has  provided  significant  progress 
toward  the  completion  of  Task  2,  differentiation  between  novice  and  expert. 

Peg  T ransfer: 

One  of  the  metrics  that  seem  to  be  a  good  discriminator  of  skill  among  novices  is  the 
total  pass  time  (i.e.  how  long  does  it  take  to  pass  the  peg  from  one  tool  to  another). 
Figure  7  shows  the  subject’s  total  time  vs.  the  total  pass  time.  The  vertical  line 
represents  the  48  second  proficiency  line  that  was  the  target.  The  horizontal  line 
represents  the  cutoff  that  all  subjects  met  who  were  proficient.  More  interesting  is  that 
the  subjects  who  were  not  proficient  rarely  (~4%)  met  this  horizontal  cutoff.  In  fact,  there 
were  only  6  trials  that  met  this  cutoff.  Such  an  extreme  cutoff  was  not  found  for  the  pick 
or  placement  of  peg  data.  This  data  indicate  that  passing  a  peg  may  be  a  good  leading 
indicator  of  how  well  the  subject  will  do  on  the  task.  It  may  be  possible  to  conduct  a 
training  regimen  where  the  subjects  only  practice  passing  a  single  peg  6  times  back  and 
forth  to  see  if  they  fall  under  this  horizontal  cutoff.  If  they  do,  then  it  could  mean  that 
they  are  more  likely  to  surpass  the  proficiency  mark  on  the  full  test.  This  would  in  effect 
reduce  the  amount  of  training  the  subject  would  have  to  do.  This  metric  may  be 
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representative  of  test  success  because,  as  mentioned  in  the  FLS  guidelines,  the 
purpose  of  the  peg  transfer  exercise  is  to  test  eye-hand  coordination,  bi-manual 
coordination  and  depth  perception.  Passing  encompasses  all  three  skills,  unlike  picking 
or  placing  a  peg  which  does  not  require  bimanual  coordination.  This  finding  is 
particularly  important  when  it  comes  time  to  create  a  reset  training  or  parts  training 
regimen. 
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Figure  7:  Scatter  plot  of  the  total  pass  time  vs.  total  time  for  the  peg  transfer  task. 
Subjects  who  have  not  reached  the  level  of  proficiency  rarely  (~4%)  met  the  horizontal 
cutoff  of  21.6s. 
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The  data  also  indicates  that  some  ambidexterity  is  required  (Figure  8).  This  is  an 
indication  of  how  well  the  dominant  and  non-dominant  hand  perform  the  same  subtask 
(pick,  pass,  place)  comparatively.  An  ambidexterity  difference  of  zero  indicates  perfect 
ambidexterity  (no  difference  whatsoever).  A  difference  of  ten  indicates  that  the  weaker 
side  is  10  seconds  slower.  Note  that  the  weaker  side  need  not  be  the  same  for  each  of 
the  subtask  (e.g.  dominant  hand  could  be  weaker  for  picking  a  peg  and  your  non¬ 
dominant  hand  could  be  weaker  for  placing  a  peg).  The  ambidexterity  difference  takes 
into  account  the  weaker  side  for  each  subtask.  It  appears  that  some  degree  of 
ambidexterity  is  necessary  to  attain  proficiency  but  is  not  a  sufficient  skill  by  itself. 
Literally,  subjects  can  be  equally  slow  with  both  hands. 
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Figure  8:  Scatter  plot  of  the  ambidexterity  difference  vs.  total  time  for  the  peg  transfer 
task.  Some  degree  of  ambidexterity  is  necessary  to  attain  proficiency  but  is  not  a 
sufficient  skill  by  itself. 


Task  Refinement: 

We  have  also  added  an  additional  adjacent  task  to  the  standard  FLS  known  as  a 
secondary  distraction.  In  this  scenario,  the  subject’s  primary  task  is  to  perform  the  FLS 
task.  Their  secondary  task  is  to  perform  a  pattern  recognition  task.  This  is  intended  to 
measure  the  extent  of  automaticity  of  task  performance,  the  ability  to  successfully 
perform  the  task  without  full  attention,  that  FLS  metrics  do  not  currently  measure. 
Automatic  movements  will  continue  on  task,  despite  distractions.  The  degree  of 
automaticity  in  the  performance  of  a  task  may  translate  into  the  real-world  OR  better 
than  primary  measures  such  as  speed.  Measuring  automaticity  may  also  be  useful  in 
the  evaluation  of  skill  degradation.  Secondary  skills  may  degrade  before  or  to  a  greater 
extent  than  primary  skills,  when  measured  on  a  simple  task  such  as  peg  transfer. 


The  pattern  recognition  involves  looking  at  an  adjacent  window/screen  and  recognizing 
when  a  circle  appears  3  consecutive  times  on  one  side  of  the  window  (Figure  9).  The 
subject  presses  a  foot  pedal  when  this  occurs.  The  circle  appears  every  second  for  0.3 
seconds  at  a  time.  The  task  is  relatively  simple  to  perform  by  itself.  The  task  requires 
constant  attention  with  respect  to  the  secondary  window  while  performing  the  primary 
task.  If  the  subject’s  entire  attentional  resources  are  devoted  to  the  primary  task,  the 
subject  will  most  likely  fail  the  secondary  task.  Stefanidis  et  al.  (Do  Novices  Display 
Automaticity  During  Simulator  Training?  The  Journal  of  American  Surgery  2008  195:  210-213.)  has 
indicated  that  this  is  a  possible  metric  to  distinguish  between  novices  and  experts  that 
have  the  same  trial  times. 


Charles  Shanley,  MD  -  page  12 


Figure  9:  Screen  shot  of  the  peg  transfer  task  along  with  the  secondary  distraction  task 
(pattern  recognition).  The  circle  appears  at  either  the  top  or  bottom  of  the  window. 
Subjects  have  to  recognize  when  it  appears  on  one  side  three  consecutive  times  while 
performing  the  primary  task  on  the  left  hand  window. 

Another  metric  that  might  be  predictive  of  how  subject  success  on  a  secondary  task  is 
the  parallel  ratio.  This  is  a  measure  of  how  much  close  the  pick  and  placement  of  pegs 
occur.  Ideally,  the  subject  would  be  able  to  pick  and  place  pegs  at  the  same  time.  This 
requires  splitting  your  attention  between  two  tasks  simultaneously  (picking  and  placing  a 
peg)  similar  to  performing  a  task  with  a  secondary  distraction. 

There  were  two  main  strategies  that  subjects  employed  in  trials  currently  analyzed.  One 
was  working  in  series,  meaning  that  they  do  a  single  subtask  (pick,  pass  and  place  in 
that  order)  one  at  a  time.  Ideally,  this  would  result  in  a  parallel  ratio  of  2.  The  other 
strategy  was  to  work  in  parallel  meaning  that  you  are  trying  to  pick  and  place  pegs 
concurrently.  Ideally,  this  would  result  in  a  parallel  ratio  of  1. 

Subjects  who  were  proficient  spanned  a  wide  range  of  parallel  ratio  indicating  that  both 
strategies  could  be  used  successfully  to  complete  the  task  (Figure  10).  More 
importantly,  employing  a  parallel  ratio  does  not  equal  success  as  the  Wayne  State  non¬ 
proficient  subjects  (red  squares)  got  very  low  parallel  ratios  but  no  proficient  times.  This 
indicates  that  the  subject  was  consciously  trying  to  work  in  parallel  but  was  not 
sufficiently  fast  enough  to  reach  the  goal  using  this  strategy.  However,  parallel  ratio 
could  predict  how  subjects  fare  on  the  same  task  with  the  secondary  distraction  since  it 
requires  splitting  your  attention  between  two  windows  or  screens. 
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Figure  10:  Scatter  plot  of  the  parallel  ratio  vs.  total  time  for  the  peg  transfer  task.  The 
degree  of  parallelism  seems  to  have  no  bearing  on  proficiency  but  it  might  on  the 
secondary  task  performance  due  to  the  splitting  of  attentional  resources  required  for 
doing  tasks  in  parallel. 

Circle  Cutting  Task: 

For  the  circle  cutting  task,  preliminary  data  from  novices  has  been  analyzed  by  training 
day.  Figure  1 1  shows  the  learning  curve  broken  down  by  training  day  for  one  subject. 
The  mean  time  for  the  day  is  also  shown. 


a  Beaumont  Proficient 
WSU  Proficient 

■  Beaumont  Non-Proficient 

■  WSU  Non-Proficient 


A- 


Figure  1 1:  Learning  curve  for  one  subject  broken  down  by  their  two  training  days.  The 
thick  black  horizontal  lines  indicates  the  mean  trial  time  for  that  day. 
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TASK  3:  Describe  a  methodology  and  provide  preliminary  data  to  document 
individual-specific  laparoscopic  skill  deterioration  during  prolonged  absence 

There  have  been  a  number  of  logistical  challenges  relating  to  navigating  the  multiple 
IRBs  required  to  complete  this  study.  Moreover,  technology  development  time  and 
funding  limitations  have  limited  our  ability  to  explore  the  impact  of  prolonged  absence 
on  skill  degradation  on  subjects  who  have  reached  proficiency.  Moreover,  this 
preliminary  work  demonstrates  that  the  impact  of  automaticity  on  skill  retention  needs 
further  exploration.  In  terms  of  surgeon  readiness,  this  implies  that  the  most 
appropriate  test  population  to  accurately  measure  the  impact  of  prolonged  absence  on 
skill  degradation  is  a  population  of  military  surgeons  pre-  and  post-deployment. 
Collecting  the  preliminary  data  for  later  comparison  with  post-deployment  evaluations  is 
beyond  the  constraints  of  funding  and  duration  of  this  project  and  would  have  to  be  the 
subject  of  future  trials. 


TASK  4:  Describe  a  methodology  to  employ  augmented  reality  technologies  to 
support  individual-specific  retraining  of  basic  laparoscopic  skills  following 
prolonged  absence 


Task  4  involves  the  translation  of  surgical  assessment  to  the  training  of  the  surgeon. 
Utilizing  advanced  augmented  reality  techniques  first  developed  to  aid  astronauts  to 
learn  and  perform  complex  tasks  in  a  difficult  environment,  we  have  implemented  a 
tracking,  assessment  and  augmented  reality  guidance  system  for  training.  Since  we 
know  the  position  and  orientation  of  the  tools  (from  the  magnetic  tracker)  and  position 
and  orientation  of  the  camera  (fixed)  then  we  can  display  augmented  reality  cues  on  the 
monitor  to  help  the  subject  learn.  Cues 
can  be  in  the  form  of  coordinates,  or  text 
or  where  to  go  next.  We  have 
demonstrated  the  ability  to  place  any 
simulated  3D  graphics  object  within  the 
view  of  the  FLS  camera. 


Augmented  reality  provides  the  most 
clinical  utility  when  it  can  be  combined 
with  tasks  such  as  intracorporeal 
suturing.  Intracorporeal  suturing  is  a 
task  that  is  difficult  to  master  and  rarely 
practiced,  even  by  surgeons  with 
significant  laparoscopic  experience. 
Learning  intracorporeal  suturing  would 
provide  significant  clinical  value  to  these 
surgeons,  and  is  very  suited  to 
instruction  and  practice  using 
simulation.  This  task  is  the  next  we  have 
automated  in  our  sensor. 


Figure  12:  Steps  in  the  hand-tie  method  of 
forming  an  intracorporeal  knot  involve 
several  instrument  transfers.  Reprinted 
from  Murphy,  DL,  Annals  of  Surgery  Vol. 
234,  No.  5,  607-612. 
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As  shown  in  the  Figure  12,  intracorporeal  knot  involves  a  complex  set  of  motions  and 
hand-eye  coordination  skills.  The  motion  analysis  software  for  intracorporeal  suturing 
analyzes  the  16  different  ways  to  place  a  square  knot  using  laparoscopic  intracorporeal 
techniques  and  the  5  to  10  different  general  motions  involved  for  each  knot 

Intracorporeal  knot  tying  motions  are  broken  down  within  the  vector  tracking  algorithm 
into  repeated  half-knots  with  five  major  components  in  each  half-knot.  The  basic 
motions  are  (A)  grasp  suture  end,  (B)  crossover,  (C)  rotate  grasping  instrument  in  the 
correct  direction  (D)  grasp  suture  tail  and  pull  through,  and  (E)  tighten.  These  motions 
can  be  tracked  and  broken  up  into  a  series  of  integrated  kinetic  event  vectors.  Ten 
motions  are  required  for  a  complete  square  knot:  4  grasping  motions,  2  rotations,  2 
crossovers,  and  2  tightening  motions.  Motion  parameters  for  each  knot-tying  event  are 
traced  utilizing  the  three-dimensional  motion  tracking  system.  For  each  motion,  an 
analysis  of  motion  constraints  is  developed. 


Grasping  motion  with  left 
instrument 


B 


Crossover  motion  of  right 
instrument 


Clockwise  rotation  of  right 
instrument,  wrapping  suture 


Grasping  motion  with  right  instrument 
to  capture  suture  tail 


E 


Pulling  motion  with  both  instruments  to  tighten  half-knot 

Figure  13:  Piecewise  motion  vectors  for  intracorporeal  half-knot  tying  are  shown  in  blue. 
Suture  (grey)  and  simulated  vessel  (red)  are  shown  for  clarity. 
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The  parameters  resulting  in  a  s  uccessful  knot  are  analyzed  based  on  tim  e  and  the 
effectiveness  of  the  k  not.  Delayed  or  improper  knot  tying  is  anal  yzed  step  by  step  t  o 
determine  the  ‘weak  link’  in  the  techni  que.  Repeated  knots  are  analyzed  and  three- 
dimensional  motion  represent  ing  successful  technique  is  generated  wit  h  boundary 
conditions  established.  The  centroid  of  the  piecewise  motion  is  determined  as  best 
practice  motion.  The  five  piecewise  vector  motions  that  are  combined  fo  r  a  completed 
half-knot  are  shown  in  Figure  13. 

Training  for  intracorporeal  knot  tying  is  ideally  suited  to  demonstrate  the  ability  of 
augmented  reality  to  gui  de  the  novice  with  coordinate  cues  learned  from  best  practice 
techniques  of  experts.  The  augmented  realit  y  algorithm  will  integrate  motion  tracking 
during  novice  performance  with  t  he  best  practice  motion  analysis  using  the  instrument 
path  data  from  expert  surgeons.  This  overla  y  will  help  guide  the  novice  in  correct  and 
efficient  instrument  movement  by  providi  ng  a  visual  and/or  auditory  cue  when  the  path 
of  their  instruments  deviate  bey  ond  the  ex  pert  pathway  overlay.  A  predicted  motion 
boundary  analysis  is  formed  to  visually  guide  the  motion  within  acceptable  parameters. 
When  the  novice  surgeons’  technique  is  within  the  proper  par  ameters  the  cues  in  the 
coordinate  system  show  a  blue  color.  De  viation  from  the  b  est  practice  boundar  y 
conditions  results  in  a  change  in  the  coordinate  cues  to  red. 

Toward  this  end  we  have  developed  an  algori  thm  that  transfers  the  motion  kinetics  for 
intracorporeal  knot  tying  to  an  augmented  reality  platform.  This  software  assesses  the 
tracking  motion  in  each  trial;  successful  atte  mpts  are  registered  as  positive,  and  out  of 
bounds  motions  are  assessed  as  negative  outcomes.  This  a  Igorithm  will  be  used  to 
gather  data  to  set  parameters  of  correct  motion  for  the  task.  In  addition  the  tracking  will 
include  timing  as  an  assessment  of  effi  ciency.  When  combined,  a  best  practic  e 
guidance  parameter  will  be  established  for  real  time  feedback  on  motion  kinetics.  A  flow 
diagram  of  this  algorithm  is  outlined  in  Figure  14  below. 


I  ntraco rp  o rial  K  not 
Tracking  and 
Augmented  Reality 


Develop  tracking 
Algorithm 


Write  code  for 
tracking 


Augmented  Reality 

Write  Feedback 
Algorithm 

1 - 

Integrate 

Augmented  Reality 
Feedback 

Perform  Tracking 
ofSquare  Knot 


Figure  14: 
Flow  Diagram 
of 

laparoscopic 
intracorporeal 
knot  task 
used  for 
tracking  and 
augmented 
reality 
feedback. 
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In  order  provide  student  feed  back  and  analysis  of  the  intracorporeal  knot  tying  we  hav  e 
developed  a  learning  neural  network  algorithm. 

The  software  tracks  the  motion  and  compares  the  best  vector  orientation  for  proper  and 
efficient  knot  tying.  The  tracking  data  from  each  trial  and  analysis  of  the  resulting  knot 
is  input  into  the  algorithm  to  determine  the  kinetic  motion  and  axial  quadrants  that  work 
or  fail  to  work.  Analyses  of  failed  attempts  are  compared  to  the  piecewise  motion  of 
successful  attempts  and  a  generalized  coordinate  system  is  established  for  each  portion 
of  the  motion.  Variation  outside  the  boundary  of  the  allowed  motion  generates  an  alarm 
in  the  augmented  reality  program  via  a  change  in  color  in  the  coordinate  overlay.  This 
is  combined  with  average  time  to  tie  and  a  best  practice  path  is  learned.  A  virtual  avatar 
of  the  grasp  and  motion  can  also  be  overlaid  in  the  visual  field  for  proper  training.  The 
logic  diagram  for  this  is  outlined  in  Figure  15. 


Analysis 

Software 


Kinetic  motion  resulted 
in  proper  knot 


Kinetic  motion  results  for 
improper  knot  or  failure 


Performed  under 
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1  J 
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Sucessful  knot  tying 


In  range  specification 
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Figure  15:  Logic  diagram  for  analysis  software  for  motion  analysis  and  augmented  reality 
information  input. 

An  additional  camera  system  was  added  to  allow  for  three-dimensional  visual  tracking 
as  related  to  the  coordinate  tracking  via  magnetic  sensor  position  location.  Software  for 
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image  analysis  tracking  was  integrated  with  the  position  tracking.  The  software  allowed 
edge  detection  by  machine  vision  for  automated  position  location.  A  tubular  phantom 
was  added  for  a  standard  knot  tying  exercise.  The  optical  system  and  magnetic 
tracking  position  sensing  system  was  calibrated  utilizing  known  location  touch  points. 
Additional  software  was  implemented  to  indicate  differences  in  left  hand  and  right  hand 
proficiency. 

The  most  recent  work  on  this  project  focused  on  developing  error  analysis  software 
capable  of  correlating  kinetic  motion  tracking  with  errors,  failures  and  delays  in  knot 
tying.  The  software  also  performs  a  differential  analysis  between  speed  and  error  rate 
for  right  hand  verses  left  hand  performance.  This  software  was  used  to  analyze  50  test 

cases.  The  scatter  plot  with 
best  practice  analysis  overlay  is 
shown  in  Figure  16a.  In  each 
case  errors  were  identified 
visually  and  translated  to  the 
error  analysis  software.  For 
example,  the  left  hand  and  right 
hand  motion  was  broken  up  into 
five  piecewise  vector  motions. 
The  best  practice  best  time 
example  was  used  as  a 
standard.  Motion  was 
compared  to  the  best  practice 
motion  at  each  of  the  five  vector 
motions.  Figure  16b  shows  five 
separate  time  segments  of  the 
best  practice  trial  corresponding 
to  the  five  vector  motions. 
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Figure  16:  a)  Scatter  plot  of  50 
trial  cases  for  intracorporeal 
laparoscopic  knot  tying. 

Upper:  left  hand  motions. 
Lower:  right  hand  motions. 
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b)  Best  practice  example  average,  over  5  time  segments,  for  translation  into  augmented 
reality  guidance.  Left  hand  data  is  shown  in  red,  right  hand  data  shown  in  blue. 

Both  variations  of  each  vector  motion  and  time  to  complete  were  compared  to  the  best 
practice  example.  All  successful  knot  tying  motions  were  compared  to  best  practice 
example  to  determine  allowable  variability  in  each  piecewise  vector  motion  to  be  used  in 
the  augmented  reality  overlay.  The  edge  detection  path  shown  in  Figure  13  represents 
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the  virtual  reality  overlay  guidance  path  for  each  of  the  five  stepwise  vector  motions  in 
knot  tying.  This  overlay  is  used  for  training  as  a  best  practice  and  for  error-detection. 


Figure  1 7:  Edge  Detection  showing  path  for  augmented  reality  best  practice.  Similar  to 
Figure  16  above,  the  left  hand  data  is  shown  in  red  and  right  hand  data  is  shown  in  blue. 
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KEY  RESEARCH  ACCOMPLISHMENTS 


•  Determination  of  surgical  skills  to  be  tested  based  on  the  FLS  curriculum 

•  Development  of  a  prototype  simulator  and  instrumentation  with  magnetic  sensors 

•  Software  development  for  the  FLS  peg  transfer  and  pattern  cutting  tasks  for  error 
detection  and  performance  metrics  capture 

•  Creation  of  new  objective  metrics  for  the  FLS  peg  transfer  and  pattern  cutting  tasks 

•  Skills  task  refinement  including  secondary  distracters 

•  Pilot  testing  and  system  verification 

•  Expert  and  Novice  subject  testing  and  data  analysis 

•  Refinement  of  instrumented  simulator  setup  to  incorporate  additional  cameras  for 
3D  data  collection 

•  Development  of  an  integrated  algorithm  to  transfer  FLS  kinetics  for  intracorporeal 
knot  tying  to  augmented  reality 

•  Development  of  error  tracking  analysis  software  for  feedback  with  augmented 
reality. 

•  Integration  of  image  tracking  hardware,  augmented  reality  overlay  and  error 
detection  algorithm 

•  Refinement  of  data  reporting  software 
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Hospital. 
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Lockheed  Martin  Corporation. 
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CONCLUSIONS 


Creation  of  an  integrated  sensing  system  with  laparoscopic  surgical  tasks  based  on  the 
FLS  curriculum,  and  generation  of  task  metrics  to  evaluate  skill  proficiency  has  been 
accomplished.  Analysis  of  data  on  peg  transfer  and  circle  cutting  tasks  resulted  in 
enhancement  of  the  skills  testing  environment.  We  anticipate  that  the  data  collected 
with  this  new  system  will  allow  refined  analysis  of  FLS  tasks.  The  system  will  allow  the 
users  and  evaluators  to  understand  particular  task  deficiencies  at  a  more  detailed  level. 
This  refined  analysis  will  also  allow  the  system  to  distinguish  novice  from  expert 
subjects  based  on  performance  measures,  and  also  determine  any  degradations  in  skill 
levels  as  compared  to  previous  attempts  (i.e.  prolonged  absence). 

There  has  also  been  significant  progress  toward  implementing  a  virtual  reality  overlay 
for  training  and  evaluation  of  intracorporeal  knot  tying  skills.  A  computer  algorithm  was 
developed  to  assess  data  from  the  FLS  to  track  the  intracorporeal  knot  tying  as  a 
complex  skill,  with  error  tracking  and  subject  feedback.  The  goal  is  to  create  teach  the 
algorithm  best  practices,  with  variations,  for  intracorporeal  knot  tying  and  translate  that 
to  an  augmented  reality  enhanced  FLS  trainer.  This  would  allow  for  real  time  visual 
feedback  and  guidance  during  the  learning  process  and  assessment  of  skill. 


Significance: 

•  The  overarching  goal  of  this  research  is  to  improve  military  surgeon  readiness  and 
enhance  patient  safety.  We  believe  this  novel  technology  provides  an  objective, 
data-driven  platform  upon  which  complex,  proficiency-based  technical  skills  can  be 
assessed,  monitored  and  improved  over  time.  This  platform  will  facilitate  the 
validation  of  individual  performance  metrics  that  can  be  used  to  monitor 
proficiency.  It  will  also  facilitate  the  development  of  curricula  to  enhance  individual 
performance,  promote  maintenance  of  proficiency  and  to  facilitate  specific 
retraining  following  periods  of  prolonged  absence  from  surgical  practice. 

•  The  development  of  the  motion  kinetics  assessment  from  the  FLS  tracking  system 
has  been  completed  as  a  major  step  toward  integrating  augmented  reality  feedback 
and  guidance.  Successful  integration  of  an  augmented  reality  overlay  with  our 
instrumented  simulator  provides  proof  of  concept  that  we  can  define  an  envelope  of 
performance  based  on  the  efficiency  of  expert  motions,  and  has  implications  for 
both  skills  degradation  evaluation  and  novice  training. 

•  Similar  techniques  may  be  applied  to  more  complex  tasks,  such  as  surgical 
subprocedures  like  bowel  anastomosis.  Several  simulation  centers  are  applying 
simulation  to  specific  technical  skill  accomplishments.  (For  example,  difficult  airway 
management.) 

•  Very  little  work  is  being  performed  in  performance  validation  of  skills,  integration  of 
technical  and  cognitive  skills,  and  the  integration  of  team  skills 
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•  As  these  performance  metrics  are  established  there  will  be  a  tremendous  demand 
for  performance-based  certification  of  healthcare  professionals.  Markets  would 
include: 

o  Certification  organizations 
o  Educational  institutions 

o  Ongoing  requirements  for  proficiency  of  healthcare  professionals 
(Privileges,  etc.) 
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